Explanation: https://link.springer.com/chapter/10.1007/978-3-030-29736-7_5

Learning data

LIFE Data Summary: Top 5
Student Session KC Try Time Feedback Correct Opportunity Gap Gap_Type Group Cycle Session_Complete Complete_Plays SRL Time.Scale Prev_Success Prev_Failure Age Experience Cadre Level ETAT Region Correct_First_Try
S1 2019-05-07 11:06:08 KC1 1 32.122 Essential 0 0 -4.60517 None Control 1 0 0 NA 1.076694 0 0 NA NA NA NA True Western Asia 0
S1 2019-05-07 11:06:08 KC1 2 11.688 None 1 1 -4.60517 None Control 1 0 0 NA -0.213432 0 1 NA NA NA NA True Western Asia 0
S2 2019-04-07 13:09:03 KC1 1 39.471 None 1 0 -4.60517 None Experiment 1 0 2 NA 1.540682 0 0 NA NA NA NA False Sub-Saharan Africa 1
S2 2019-04-07 13:09:03 KC2 1 91.466 Reflective 0 0 -4.60517 None Experiment 1 0 2 NA 4.823450 0 0 NA NA NA NA False Sub-Saharan Africa 0
S2 2019-04-07 13:09:03 KC2 2 36.273 Detailed 0 1 -4.60517 None Experiment 1 0 2 NA 1.338772 0 1 NA NA NA NA False Sub-Saharan Africa 0
S2 2019-04-07 13:09:03 KC2 3 19.065 None 1 2 -4.60517 None Experiment 1 0 2 NA 0.252324 0 2 NA NA NA NA False Sub-Saharan Africa 0


Summary statistics

Feedback Summary
Levels Incomplete Complete All
None 1541 (13.66%) 9743 (86.34%) 11284 (60.21%)
Essential 622 (26.49%) 1726 (73.51%) 2348 (12.53%)
Reflective 264 (23.51%) 859 (76.49%) 1123 (5.99%)
Detailed 1353 (33.95%) 2632 (66.05%) 3985 (21.26%)
Spacing summary
Levels Incomplete Complete All
None 256 (44.76%) 316 (55.24%) 572 (37.91%)
<= 1 Hour 156 (30%) 364 (70%) 520 (34.46%)
<= 1 Day 52 (28.42%) 131 (71.58%) 183 (12.13%)
<= 1 Week 37 (30.83%) 83 (69.17%) 120 (7.95%)
<= 1 Month 27 (36.49%) 47 (63.51%) 74 (4.9%)
> 1 Month 21 (52.5%) 19 (47.5%) 40 (2.65%)

69.93% of learners had at least one complete learning session


The Bayesian Knowledge Tracing (BKT) model assumes the latent states to be independence among the difference skills in the learning module. Even within an individual skill, BKT assumes the probability of the next learning outcome to depend only on the latest previous outcome (The Markov assumption). Additionally BKT assumes that:

  1. Forgetting does not occur
    1. Its typical implementation does not allow for learners to have different learning rates
    2. It assumes that all students have the same probability of knowing a particular skill at their first opportunity (Pardos and Heffernan, 2010)
    3. It suffers from the problem of multiple global maxima when trying to estimate model parameters (Gong et al., 2011).
    4. While multiple-restarts estimation methods like Markov Chains Monte-Carlo estimations can help minimise local maxima challenges in estimating BKT model parameters, it is hard to eradicate the identifiability problem where given the same model structure and same data, there are multiple (differing) sets of parameter values that fit the data equally well (Gong et al., 2011). This is because in probabilistic terms, to find the global solution requires the model to iterate over all possible learner sequences. However, iterating over all learner sequences of varying lengths in order to maximize the objective function makes the number of iterations exponential to the size of the learning data, rendering it practically intractable.

To address some of the shortcomings of BKT models, Learning Factors Analysis (LFA) and their different modalities such as Performance Factors Models (PFMs) and Additive Factor Models(AFMs) have been proposed (Cen et al., 2006, Cen et al., 2008, Pavlik Jr et al., 2009). They model student knowledge states using logistic regression models in order to deal with the incorporation of multiple skills while estimating student ability into the model. They exploit the number of successes or failures of a learner’s attempt at a KC to predict whether the learner has acquired understanding about the KC.

More details on the application of these models for this dataset can be found here: https://link.springer.com/chapter/10.1007/978-3-030-29736-7_5

Cross-validation of model performance in predicting outcome on next step

AFM Accuracy: 0.689 (0.674 - 0.696) AUC: 0.748 (0.737 - 0.764)

PFM Accuracy: 0.767 (0.751 - 0.776) AUC: 0.85 (0.841 - 0.856)


PFM model diagnostics

The ‘Student’ variable accounts for 27.01% of the stochastic variation in the PFM model

L. Zeger, K. Y. Liang, and P. S. Albert. Models for longitudinal data: a generalized estimating equation approach. Biometrics, 44: 1049-1060 1988


Minimum (Base) model


Rash tests

Person-Item Map

IRT model fit

Andersen LR-test: LR-value: 4.731 Chi-square df: 9 p-value: 0.857

Infit T-statistic

Pathway maps are useful for identifying misfitting items or misfitting persons. Items or people should ideally have a infit t-statistic lying between about -2 and +2, and these values are marked

Plot visualizing the item characteristic curves for LIFE

Item difficulty

Compare Base and Detailed PFM model

The ‘Student’ variable accounts for 18.79% of the stochastic variation in the base PFM model

The ‘Student’ variable accounts for 12.18% of the stochastic variation in the detailed PFM model

AFM Brier Score: 0.3771

PFM Brier Score: 0.3099

  Correct_First_Try Correct_First_Try
Predictors Odds Ratios CI p Odds Ratios CI p
(Intercept) 0.99 0.81 – 1.19 0.877 1.57 1.19 – 2.05 0.001
KC [KC2] 0.44 0.37 – 0.53 <0.001 0.36 0.28 – 0.46 <0.001
KC [KC3] 1.09 0.88 – 1.35 0.423 0.83 0.60 – 1.13 0.238
KC [KC4] 0.69 0.58 – 0.83 <0.001 0.54 0.42 – 0.70 <0.001
KC [KC5] 0.55 0.46 – 0.66 <0.001 0.42 0.33 – 0.53 <0.001
KC [KC6] 1.13 0.91 – 1.41 0.251 1.24 0.93 – 1.65 0.142
KC [KC7] 0.66 0.55 – 0.78 <0.001 0.50 0.40 – 0.64 <0.001
KC [KC8] 0.74 0.62 – 0.89 0.002 0.62 0.48 – 0.80 <0.001
KC [KC9] 1.09 0.88 – 1.33 0.432 0.88 0.66 – 1.16 0.369
KC [KC10] 1.01 0.82 – 1.25 0.901 0.95 0.72 – 1.24 0.704
Group [Experiment] 1.08 0.90 – 1.28 0.410 0.91 0.72 – 1.14 0.406
ETAT [True] 0.99 0.83 – 1.17 0.907 1.09 0.86 – 1.38 0.488
KC [KC1] :
Prev_Success.Scale 		9.96 7.95 – 12.48 <0.001 3.67 2.86 – 4.70 <0.001
KC [KC2] :
Prev_Success.Scale 		8.60 6.99 – 10.58 <0.001 4.79 3.76 – 6.09 <0.001
KC [KC3] :
Prev_Success.Scale 		8.42 6.62 – 10.70 <0.001 4.15 3.16 – 5.45 <0.001
KC [KC4] :
Prev_Success.Scale 		8.29 6.66 – 10.32 <0.001 5.21 3.96 – 6.84 <0.001
KC [KC5] :
Prev_Success.Scale 		8.97 7.21 – 11.16 <0.001 5.59 4.21 – 7.41 <0.001
KC [KC6] :
Prev_Success.Scale 		5.19 4.11 – 6.55 <0.001 2.44 1.90 – 3.13 <0.001
KC [KC7] :
Prev_Success.Scale 		5.71 4.69 – 6.96 <0.001 3.50 2.77 – 4.43 <0.001
KC [KC8] :
Prev_Success.Scale 		5.37 4.33 – 6.67 <0.001 3.13 2.42 – 4.06 <0.001
KC [KC9] :
Prev_Success.Scale 		19.09 13.65 – 26.69 <0.001 12.63 7.93 – 20.12 <0.001
KC [KC10] :
Prev_Success.Scale 		5.12 4.02 – 6.53 <0.001 2.67 2.05 – 3.48 <0.001
KC [KC1] :
Prev_Failure.Scale 		0.26 0.20 – 0.33 <0.001 0.38 0.27 – 0.54 <0.001
KC [KC2] :
Prev_Failure.Scale 		0.55 0.45 – 0.67 <0.001 0.47 0.36 – 0.62 <0.001
KC [KC3] :
Prev_Failure.Scale 		0.27 0.20 – 0.38 <0.001 0.17 0.10 – 0.28 <0.001
KC [KC4] :
Prev_Failure.Scale 		0.33 0.25 – 0.44 <0.001 0.27 0.18 – 0.40 <0.001
KC [KC5] :
Prev_Failure.Scale 		0.53 0.43 – 0.66 <0.001 0.48 0.35 – 0.65 <0.001
KC [KC6] :
Prev_Failure.Scale 		0.24 0.18 – 0.33 <0.001 0.31 0.21 – 0.44 <0.001
KC [KC7] :
Prev_Failure.Scale 		0.48 0.39 – 0.59 <0.001 0.51 0.39 – 0.67 <0.001
KC [KC8] :
Prev_Failure.Scale 		0.35 0.27 – 0.45 <0.001 0.41 0.30 – 0.56 <0.001
KC [KC9] :
Prev_Failure.Scale 		0.28 0.22 – 0.37 <0.001 0.18 0.12 – 0.27 <0.001
KC [KC10] :
Prev_Failure.Scale 		0.21 0.15 – 0.27 <0.001 0.31 0.22 – 0.44 <0.001
Opportunity.Scale :
FeedbackNone 		0.51 0.45 – 0.59 <0.001 1.17 0.98 – 1.41 0.089
Opportunity.Scale :
FeedbackEssential 		1.27 1.08 – 1.51 0.005 0.86 0.69 – 1.08 0.193
Opportunity.Scale :
FeedbackReflective 		5.26 4.04 – 6.85 <0.001 4.13 2.99 – 5.70 <0.001
Opportunity.Scale :
Time.Scale 		0.89 0.84 – 0.93 <0.001 1.00 0.94 – 1.07 0.901
Opportunity.Scale :
Gap_Type<= 1 Hour 		1.09 0.94 – 1.27 0.244 0.80 0.65 – 0.99 0.039
Opportunity.Scale :
Gap_Type<= 1 Day 		0.61 0.51 – 0.75 <0.001 0.42 0.33 – 0.53 <0.001
Opportunity.Scale :
Gap_Type<= 1 Week 		0.46 0.38 – 0.56 <0.001 0.32 0.24 – 0.41 <0.001
Opportunity.Scale :
Gap_Type<= 1 Month 		0.20 0.16 – 0.26 <0.001 0.20 0.15 – 0.27 <0.001
Opportunity.Scale :
Gap_Type> 1 Month 		0.24 0.16 – 0.34 <0.001 0.36 0.24 – 0.53 <0.001
Opportunity.Scale :
SRLLow SRL profile 		1.98 1.15 – 3.41 0.013
Opportunity.Scale :
SRLAverage SRL profile 		1.70 1.39 – 2.08 <0.001
Opportunity.Scale :
SRLAbove Average SRL
profile 		1.08 0.93 – 1.24 0.331
Time.Scale : SRLLow SRL
profile 		0.91 0.56 – 1.48 0.711
Time.Scale : SRLAverage
SRL profile 		1.05 0.90 – 1.22 0.532
Time.Scale : SRLAbove
Average SRL profile 		0.98 0.89 – 1.08 0.682
Random Effects
σ2 3.29 3.29
τ00 0.67 Student 0.40 Student
ICC 0.17 0.11
N 572 Student 164 Student
Observations 18120 8342
Marginal R2 / Conditional R2 0.547 / 0.624 0.529 / 0.580